Human Action Recognition Using Deep Learning¶
Description¶
Human activity recognition plays a significant role in human-to-human interaction andinterpersonal relations. Because it provides information about the identity of a person, theirpersonality, and psychological state, it is difficult to extract. The human ability to recognizeanother person's activities is one of the main subjects of study of the scientific areas ofcomputer vision and machine learning. As a result of this research, many applications,including video surveillance systems, human-computer interaction, and robotics for humanbehaviour characterization, require a multiple activity recognition system. In image and videoanalysis, human activity recognition is an important research direction. In the past, a largenumber of papers have been published on human activity recognition in video and imagesequences. In this paper, we provide a comprehensive survey of the recent development ofthe techniques, including methods, systems, and quantitative evaluation of the performance ofhuman activity recognition. The experimental results show that our method can significantlyimprove classification, interpretation, and retrieval performance for the video images. Thenovelty of this paper is twofold. First to capture the video images of human. Secondly, toidentify the different types of action performed by human.¶
The uniqueness of the human action shape or silhouette can be used for the human actionrecognition. Acquiring the features of human silhouette to obtained the concept of humanaction invarianceness have led to an important research in video surveillance domain. Thispaper discusses the investigation of this concept by extracting individual human actionfeatures using integration moment invariant. Experiment result have shown that human actioninvarianceness are improved with better recognition accuracy. This has verified that theintegration method of moment invariant is worth explored in recognition of human action invideo surveillance.¶
Models are being made wherein, if an image is given to the model, it can predict what theimage is about, or it can detect whether a particular object is present in the image or not.These models are known as neural networks (or artificial neural networks) which are inspiredby the structure and functionality of a human brain. Deep learning, a subfield of Machinelearning is the study of these neural networks and over the time, a number of variations ofthese networks have been implemented for a variety of different problems. This project usesdeep learning for Video Recognition - given a set of labelled videos, train a model so that itcan give a label/prediction for a new video. Here, the label might represent what is beingperformed in the video, or what the video is about¶
2.Deep learning¶
2.2 Performance Metric ---> Accuracy¶
3.1 Dataset Link
In [ ]:
Unzip the dataset from drive¶
Dataset Link : https://drive.google.com/file/d/1bjKIV1hgLg_2HWK6bGhBBVOib2BqdBmG/view?usp=sharing¶
In [ ]:
!pip install patool #instead of !unzip command we are using patool because it will unzip the data much faster compare to !unzip command
Collecting patool
Downloading patool-1.12-py2.py3-none-any.whl (77 kB)
|████████████████████████████████| 77 kB 4.8 MB/s
Installing collected packages: patool
Successfully installed patool-1.12
In [ ]:
import patoolib
dataset_path = '/content/drive/MyDrive/Technocolabs_Internship_Project/Dataset/train_images.7z'
patoolib.extract_archive(dataset_path, outdir='/content/Dataset')
patool: Extracting /content/drive/MyDrive/Technocolabs_Internship_Project/Dataset/train_images.7z ... patool: running /usr/bin/7z x -o/content/Dataset -- /content/drive/MyDrive/Technocolabs_Internship_Project/Dataset/train_images.7z patool: ... /content/drive/MyDrive/Technocolabs_Internship_Project/Dataset/train_images.7z extracted to `/content/Dataset'.
Out[ ]:
'/content/Dataset'
In [ ]:
!pip install pafy youtube-dl moviepy
Collecting pafy
Downloading pafy-0.5.5-py2.py3-none-any.whl (35 kB)
Collecting youtube-dl
Downloading youtube_dl-2021.6.6-py2.py3-none-any.whl (1.9 MB)
|████████████████████████████████| 1.9 MB 13.7 MB/s
Requirement already satisfied: moviepy in /usr/local/lib/python3.7/dist-packages (0.2.3.5)
Requirement already satisfied: decorator<5.0,>=4.0.2 in /usr/local/lib/python3.7/dist-packages (from moviepy) (4.4.2)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from moviepy) (1.19.5)
Requirement already satisfied: tqdm<5.0,>=4.11.2 in /usr/local/lib/python3.7/dist-packages (from moviepy) (4.62.3)
Requirement already satisfied: imageio<3.0,>=2.1.2 in /usr/local/lib/python3.7/dist-packages (from moviepy) (2.4.1)
Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from imageio<3.0,>=2.1.2->moviepy) (7.1.2)
Installing collected packages: youtube-dl, pafy
Successfully installed pafy-0.5.5 youtube-dl-2021.6.6
Now let's import some libraries¶
In [ ]:
# Standard Imports
import os
import cv2
import math
import pafy
import random
import seaborn
import PIL
import numpy as np
import datetime
from moviepy.editor import *
from collections import deque
import matplotlib.pyplot as plt
%matplotlib inline
# For Network
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Dropout
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import MaxPool2D
from tensorflow.keras.layers import BatchNormalization
from tensorflow.keras.layers import GlobalAveragePooling2D
from tensorflow.keras.utils import to_categorical
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras.callbacks import ReduceLROnPlateau
from tensorflow.keras.callbacks import TensorBoard
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.python.keras.utils.vis_utils import plot_model
Imageio: 'ffmpeg-linux64-v3.3.1' was not found on your computer; downloading it now. Try 1. Download from https://github.com/imageio/imageio-binaries/raw/master/ffmpeg/ffmpeg-linux64-v3.3.1 (43.8 MB) Downloading: 8192/45929032 bytes (0.0%)3514368/45929032 bytes (7.7%)7626752/45929032 bytes (16.6%)11796480/45929032 bytes (25.7%)15876096/45929032 bytes (34.6%)20062208/45929032 bytes (43.7%)24158208/45929032 bytes (52.6%)28229632/45929032 bytes (61.5%)32497664/45929032 bytes (70.8%)36380672/45929032 bytes (79.2%)40361984/45929032 bytes (87.9%)44564480/45929032 bytes (97.0%)45929032/45929032 bytes (100.0%) Done File saved as /root/.imageio/ffmpeg/ffmpeg-linux64-v3.3.1.
Dataset Path¶
In [ ]:
Dataset_dir = '/content/Dataset/train_images'
Dataset Information¶
In [ ]:
count = 0
actions_class = []
total_images = 0
for i in os.listdir(Dataset_dir):
count+=1
actions_class.append(i)
image_path = os.listdir(Dataset_dir+'/'+i)
print(f'{i} : {len(image_path)}\n')
total_images+=len(image_path)
print('Total Number of Images in Dataset is: ',total_images)
pushup : 2257 raisingeyebrows : 2954 eatingicecream : 2591 claypotterymaking : 2804 cookingoncampfire : 2892 ridingscooter : 2707 breakdancing : 2702 calligraphy : 2707 climbingarope : 2965 golfdriving : 2657 celebrating : 2364 Total Number of Images in Dataset is: 29600
3.2 Data Visualization¶
3.2.1 *Break Dancing*¶
In [ ]:
breakdancing_dir = '/content/Dataset/train_images/breakdancing'
breakdancing = os.listdir(breakdancing_dir)
plt.figure(figsize=(15,15))
for i in range(6):
plt.subplot(3, 3, i + 1)
img = plt.imread(os.path.join(breakdancing_dir, breakdancing[i]))
plt.imshow(img, cmap='gray')
plt.axis('off')
plt.tight_layout()
-------------------------------------------------------------------CalliGraphy-------------------------------------------------------------------¶
In [ ]:
calligraphy_dir = '/content/Dataset/train_images/calligraphy'
calligraphy = os.listdir(calligraphy_dir)
plt.figure(figsize=(15,15))
for i in range(6):
plt.subplot(3, 3, i + 1)
img = plt.imread(os.path.join(calligraphy_dir, calligraphy[i]))
plt.imshow(img, cmap='gray')
plt.axis('off')
plt.tight_layout()
-------------------------------------------------------------------Celebrating-------------------------------------------------------------------¶
In [ ]:
celebrating_dir = '/content/Dataset/train_images/celebrating'
celebrating = os.listdir(celebrating_dir)
plt.figure(figsize=(15,15))
for i in range(6):
plt.subplot(3, 3, i + 1)
img = plt.imread(os.path.join(celebrating_dir, celebrating[i]))
plt.imshow(img, cmap='gray')
plt.axis('off')
plt.tight_layout()
-------------------------------------------------------------------Claypotterymaking-------------------------------------------------------------------¶
In [ ]:
claypotterymaking_dir = '/content/Dataset/train_images/claypotterymaking'
claypotterymaking = os.listdir(claypotterymaking_dir)
plt.figure(figsize=(15,15))
for i in range(6):
plt.subplot(3, 3, i + 1)
img = plt.imread(os.path.join(claypotterymaking_dir, claypotterymaking[i]))
plt.imshow(img, cmap='gray')
plt.axis('off')
plt.tight_layout()
-------------------------------------------------------------------Climbingarope-------------------------------------------------------------------¶
In [ ]:
climbingarope_dir = '/content/Dataset/train_images/climbingarope'
climbingarope = os.listdir(climbingarope_dir)
plt.figure(figsize=(15,15))
for i in range(6):
plt.subplot(3, 3, i + 1)
img = plt.imread(os.path.join(climbingarope_dir, climbingarope[i]))
plt.imshow(img, cmap='gray')
plt.axis('off')
plt.tight_layout()
-------------------------------------------------------------------Cooking on campfire-------------------------------------------------------------------¶
In [ ]:
cookingoncampfire_dir = '/content/Dataset/train_images/cookingoncampfire'
cookingoncampfire = os.listdir(cookingoncampfire_dir)
plt.figure(figsize=(15,15))
for i in range(6):
plt.subplot(3, 3, i + 1)
img = plt.imread(os.path.join(cookingoncampfire_dir, cookingoncampfire[i]))
plt.imshow(img, cmap='gray')
plt.axis('off')
plt.tight_layout()
-------------------------------------------------------------------Eating Ice-cream-------------------------------------------------------------------¶
In [ ]:
eatingicecream_dir = '/content/Dataset/train_images/eatingicecream'
eatingicecream = os.listdir(eatingicecream_dir)
plt.figure(figsize=(15,15))
for i in range(6):
plt.subplot(3, 3, i + 1)
img = plt.imread(os.path.join(eatingicecream_dir, eatingicecream[i]))
plt.imshow(img, cmap='gray')
plt.axis('off')
plt.tight_layout()
-------------------------------------------------------------------Golf Driving-------------------------------------------------------------------¶
In [ ]:
golfdriving_dir = '/content/Dataset/train_images/golfdriving'
golfdriving = os.listdir(golfdriving_dir)
plt.figure(figsize=(15,15))
for i in range(6):
plt.subplot(3, 3, i + 1)
img = plt.imread(os.path.join(golfdriving_dir, golfdriving[i]))
plt.imshow(img, cmap='gray')
plt.axis('off')
plt.tight_layout()
-------------------------------------------------------------------Pushup-------------------------------------------------------------------¶
In [ ]:
pushup_dir = '/content/Dataset/train_images/pushup'
pushup = os.listdir(pushup_dir)
plt.figure(figsize=(15,15))
for i in range(6):
plt.subplot(3, 3, i + 1)
img = plt.imread(os.path.join(pushup_dir, pushup[i]))
plt.imshow(img, cmap='gray')
plt.axis('off')
plt.tight_layout()
-------------------------------------------------------------------Raising eyebrows-------------------------------------------------------------------¶
In [ ]:
raisingeyebrows_dir = '/content/Dataset/train_images/raisingeyebrows'
raisingeyebrows = os.listdir(raisingeyebrows_dir)
plt.figure(figsize=(15,15))
for i in range(6):
plt.subplot(3, 3, i + 1)
img = plt.imread(os.path.join(raisingeyebrows_dir, raisingeyebrows[i]))
plt.imshow(img, cmap='gray')
plt.axis('off')
plt.tight_layout()
-------------------------------------------------------------------Riding Scooter-------------------------------------------------------------------¶
In [ ]:
ridingscooter_dir = '/content/Dataset/train_images/ridingscooter'
ridingscooter = os.listdir(ridingscooter_dir)
plt.figure(figsize=(15,15))
for i in range(6):
plt.subplot(3, 3, i + 1)
img = plt.imread(os.path.join(ridingscooter_dir, ridingscooter[i]))
plt.imshow(img, cmap='gray')
plt.axis('off')
plt.tight_layout()
4.1 Data Augmentation¶
5.1.1 Random Erasing Method¶
In [ ]:
# import numpy as np
# def get_random_eraser(p=0.5, s_l=0.02, s_h=0.4, r_1=0.3, r_2=1/0.3, v_l=0, v_h=255, pixel_level=False):
# def eraser(input_img):
# img_h, img_w, img_c = input_img.shape
# p_1 = np.random.rand()
# if p_1 > p:
# return input_img
# while True:
# s = np.random.uniform(s_l, s_h) * img_h * img_w
# r = np.random.uniform(r_1, r_2)
# w = int(np.sqrt(s / r))
# h = int(np.sqrt(s * r))
# left = np.random.randint(0, img_w)
# top = np.random.randint(0, img_h)
# if left + w <= img_w and top + h <= img_h:
# break
# if pixel_level:
# c = np.random.uniform(v_l, v_h, (h, w, img_c))
# else:
# c = np.random.uniform(v_l, v_h)
# input_img[top:top + h, left:left + w, :] = c
# return input_img
# return eraser
In [ ]:
Batch_size = 64
Image_size = 224
classes_list = ["breakdancing","calligraphy","celebrating","claypotterymaking",
"climbingarope","cookingoncampfire","eatingicecream","golfdriving",
"pushup","raisingeyebrows","ridingscooter"]
output_size = len(classes_list)
In [ ]:
#Train:
Train_generator = tf.keras.preprocessing.image_dataset_from_directory(Dataset_dir,
labels="inferred",
batch_size=Batch_size,
color_mode="rgb",
image_size=(Image_size,Image_size),
validation_split=0.2,
subset='training',
shuffle=True,
seed=25)
#Validation:
Validation_generator = tf.keras.preprocessing.image_dataset_from_directory(Dataset_dir,
labels="inferred",
batch_size=Batch_size,
color_mode="rgb",
image_size=(Image_size,Image_size),
validation_split=0.2,
subset='validation',
shuffle=True,
seed=25)
Found 29600 files belonging to 11 classes. Using 23680 files for training. Found 29600 files belonging to 11 classes. Using 5920 files for validation.
In [ ]:
Train_generator.class_names
Out[ ]:
['breakdancing', 'calligraphy', 'celebrating', 'claypotterymaking', 'climbingarope', 'cookingoncampfire', 'eatingicecream', 'golfdriving', 'pushup', 'raisingeyebrows', 'ridingscooter']
5.Modeling¶
5.1 Model - 1 (Custom CNN)¶
In [ ]:
def human_action_recognition_model():
input_layer = tf.keras.Input(shape=(Image_size,Image_size,3))
#First-Block
conv_1 = tf.keras.layers.Conv2D(filters=64,
kernel_size=(3,3),
padding='same',
activation='relu')(input_layer)
conv_2 = tf.keras.layers.Conv2D(filters=64,
kernel_size=(3,3),
padding='same',
activation='relu')(conv_1)
maxpool_1 = tf.keras.layers.MaxPool2D(pool_size=(2,2))(conv_2)
#Second-Block
conv_3 = tf.keras.layers.Conv2D(filters=128,
kernel_size=(3,3),
padding='same',
activation='relu')(maxpool_1)
conv_4 = tf.keras.layers.Conv2D(filters=128,
kernel_size=(3,3),
padding='same',
activation='relu')(conv_3)
maxpool_2 = tf.keras.layers.MaxPool2D(pool_size=(2,2))(conv_4)
#Third-Block
conv_5 = tf.keras.layers.Conv2D(filters=256,
kernel_size=(3,3),
padding='same',
activation='relu')(maxpool_2)
conv_6 = tf.keras.layers.Conv2D(filters=256,
kernel_size=(3,3),
padding='same',
activation='relu')(conv_5)
conv_7 = tf.keras.layers.Conv2D(filters=256,
kernel_size=(3,3),
padding='same',
activation='relu')(conv_6)
maxpool_3 = tf.keras.layers.MaxPool2D(pool_size=(2,2))(conv_7)
#Forth Block
conv_8 = tf.keras.layers.Conv2D(filters=512,
kernel_size=(3,3),
padding='same',
activation='relu')(maxpool_3)
conv_9 = tf.keras.layers.Conv2D(filters=512,
kernel_size=(3,3),
padding='same',
activation='relu')(conv_8)
conv_10 = tf.keras.layers.Conv2D(filters=512,
kernel_size=(3,3),
padding='same',
activation='relu')(conv_9)
maxpool_4 = tf.keras.layers.MaxPool2D(pool_size=(2,2))(conv_10)
#Fifth Block
conv_11 = tf.keras.layers.Conv2D(filters=512,
kernel_size=(3,3),
padding='same',
activation='relu')(maxpool_4)
conv_12 = tf.keras.layers.Conv2D(filters=512,
kernel_size=(3,3),
padding='same',
activation='relu')(conv_11)
conv_13 = tf.keras.layers.Conv2D(filters=512,
kernel_size=(3,3),
padding='same',
activation='relu')(conv_12)
maxpool_5 = tf.keras.layers.MaxPool2D(pool_size=(2,2))(conv_13)
flatten_layer = tf.keras.layers.Flatten()(maxpool_5)
dense_1 = tf.keras.layers.Dense(512, activation='relu')(flatten_layer)
batchnorm_1 = tf.keras.layers.BatchNormalization()(dense_1)
dropout_1 = tf.keras.layers.Dropout(0.5)(batchnorm_1)
dense_2 = tf.keras.layers.Dense(256, activation='relu')(dropout_1)
batchnorm_2 = tf.keras.layers.BatchNormalization()(dense_2)
dropout_2 = tf.keras.layers.Dropout(0.5)(batchnorm_2)
dense_3 = tf.keras.layers.Dense(128, activation='relu')(dropout_2)
batchnorm_3 = tf.keras.layers.BatchNormalization()(dense_3)
dropout_3 = tf.keras.layers.Dropout(0.5)(batchnorm_3)
output_layer = tf.keras.layers.Dense(output_size, activation='softmax')(dropout_3)
action_model = tf.keras.Model(inputs=input_layer, outputs=output_layer)
return action_model
In [ ]:
model = human_action_recognition_model()
In [ ]:
model.summary()
Model: "model_1" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_2 (InputLayer) [(None, 224, 224, 3)] 0 _________________________________________________________________ conv2d_13 (Conv2D) (None, 224, 224, 64) 1792 _________________________________________________________________ conv2d_14 (Conv2D) (None, 224, 224, 64) 36928 _________________________________________________________________ max_pooling2d_5 (MaxPooling2 (None, 112, 112, 64) 0 _________________________________________________________________ conv2d_15 (Conv2D) (None, 112, 112, 128) 73856 _________________________________________________________________ conv2d_16 (Conv2D) (None, 112, 112, 128) 147584 _________________________________________________________________ max_pooling2d_6 (MaxPooling2 (None, 56, 56, 128) 0 _________________________________________________________________ conv2d_17 (Conv2D) (None, 56, 56, 256) 295168 _________________________________________________________________ conv2d_18 (Conv2D) (None, 56, 56, 256) 590080 _________________________________________________________________ conv2d_19 (Conv2D) (None, 56, 56, 256) 590080 _________________________________________________________________ max_pooling2d_7 (MaxPooling2 (None, 28, 28, 256) 0 _________________________________________________________________ conv2d_20 (Conv2D) (None, 28, 28, 512) 1180160 _________________________________________________________________ conv2d_21 (Conv2D) (None, 28, 28, 512) 2359808 _________________________________________________________________ conv2d_22 (Conv2D) (None, 28, 28, 512) 2359808 _________________________________________________________________ max_pooling2d_8 (MaxPooling2 (None, 14, 14, 512) 0 _________________________________________________________________ conv2d_23 (Conv2D) (None, 14, 14, 512) 2359808 _________________________________________________________________ conv2d_24 (Conv2D) (None, 14, 14, 512) 2359808 _________________________________________________________________ conv2d_25 (Conv2D) (None, 14, 14, 512) 2359808 _________________________________________________________________ max_pooling2d_9 (MaxPooling2 (None, 7, 7, 512) 0 _________________________________________________________________ flatten_1 (Flatten) (None, 25088) 0 _________________________________________________________________ dense_3 (Dense) (None, 512) 12845568 _________________________________________________________________ batch_normalization (BatchNo (None, 512) 2048 _________________________________________________________________ dropout_2 (Dropout) (None, 512) 0 _________________________________________________________________ dense_4 (Dense) (None, 256) 131328 _________________________________________________________________ batch_normalization_1 (Batch (None, 256) 1024 _________________________________________________________________ dropout_3 (Dropout) (None, 256) 0 _________________________________________________________________ dense_5 (Dense) (None, 128) 32896 _________________________________________________________________ batch_normalization_2 (Batch (None, 128) 512 _________________________________________________________________ dropout_4 (Dropout) (None, 128) 0 _________________________________________________________________ dense_6 (Dense) (None, 11) 1419 ================================================================= Total params: 27,729,483 Trainable params: 27,727,691 Non-trainable params: 1,792 _________________________________________________________________
In [ ]:
plot_model(model, show_shapes=True, show_layer_names=True, to_file='Model.png')
Out[ ]:
Compile the Model¶
In [ ]:
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001),
loss=tf.keras.losses.SparseCategoricalCrossentropy(),
metrics=['accuracy'])
Callbacks¶
In [ ]:
model_checkpoint = tf.keras.callbacks.ModelCheckpoint(filepath='/content/ModelCheckpoint/best_weights.hdf5',
save_best_only=True,
monitor='val_accuracy',
mode='max',
verbose=1)
tensorboard = tf.keras.callbacks.TensorBoard(log_dir='/content/logs')
Let's Start the Training¶
In [ ]:
history = model.fit(Train_generator,
epochs=15,
batch_size=Batch_size,
validation_data=Validation_generator,
callbacks=[model_checkpoint, tensorboard])
Epoch 1/15 370/370 [==============================] - 163s 434ms/step - loss: 2.7003 - accuracy: 0.2227 - val_loss: 3.2517 - val_accuracy: 0.1492 Epoch 00001: val_accuracy did not improve from 0.98530 Epoch 2/15 370/370 [==============================] - 161s 432ms/step - loss: 1.7523 - accuracy: 0.4519 - val_loss: 1.1789 - val_accuracy: 0.6581 Epoch 00002: val_accuracy did not improve from 0.98530 Epoch 3/15 370/370 [==============================] - 161s 432ms/step - loss: 1.0536 - accuracy: 0.6626 - val_loss: 0.6822 - val_accuracy: 0.8108 Epoch 00003: val_accuracy did not improve from 0.98530 Epoch 4/15 370/370 [==============================] - 161s 432ms/step - loss: 0.5539 - accuracy: 0.8334 - val_loss: 0.2891 - val_accuracy: 0.9373 Epoch 00004: val_accuracy did not improve from 0.98530 Epoch 5/15 370/370 [==============================] - 161s 433ms/step - loss: 0.3170 - accuracy: 0.9123 - val_loss: 0.1832 - val_accuracy: 0.9650 Epoch 00005: val_accuracy did not improve from 0.98530 Epoch 6/15 370/370 [==============================] - 161s 432ms/step - loss: 0.2009 - accuracy: 0.9495 - val_loss: 0.0782 - val_accuracy: 0.9765 Epoch 00006: val_accuracy did not improve from 0.98530 Epoch 7/15 370/370 [==============================] - 161s 432ms/step - loss: 0.2382 - accuracy: 0.9337 - val_loss: 0.1042 - val_accuracy: 0.9750 Epoch 00007: val_accuracy did not improve from 0.98530 Epoch 8/15 370/370 [==============================] - 161s 432ms/step - loss: 0.1451 - accuracy: 0.9618 - val_loss: 0.0669 - val_accuracy: 0.9806 Epoch 00008: val_accuracy did not improve from 0.98530 Epoch 9/15 370/370 [==============================] - 161s 432ms/step - loss: 0.0918 - accuracy: 0.9764 - val_loss: 0.0493 - val_accuracy: 0.9833 Epoch 00009: val_accuracy did not improve from 0.98530 Epoch 10/15 370/370 [==============================] - 161s 432ms/step - loss: 0.0736 - accuracy: 0.9796 - val_loss: 0.0681 - val_accuracy: 0.9829 Epoch 00010: val_accuracy did not improve from 0.98530 Epoch 11/15 370/370 [==============================] - 161s 432ms/step - loss: 0.0628 - accuracy: 0.9816 - val_loss: 0.0326 - val_accuracy: 0.9867 Epoch 00011: val_accuracy improved from 0.98530 to 0.98666, saving model to /content/ModelCheckpoint/best_weights.hdf5 Epoch 12/15 370/370 [==============================] - 161s 433ms/step - loss: 0.0517 - accuracy: 0.9844 - val_loss: 0.0275 - val_accuracy: 0.9887 Epoch 00012: val_accuracy improved from 0.98666 to 0.98868, saving model to /content/ModelCheckpoint/best_weights.hdf5 Epoch 13/15 370/370 [==============================] - 161s 432ms/step - loss: 0.0553 - accuracy: 0.9827 - val_loss: 0.0546 - val_accuracy: 0.9855 Epoch 00013: val_accuracy did not improve from 0.98868 Epoch 14/15 370/370 [==============================] - 161s 433ms/step - loss: 0.0512 - accuracy: 0.9842 - val_loss: 0.0262 - val_accuracy: 0.9863 Epoch 00014: val_accuracy did not improve from 0.98868 Epoch 15/15 370/370 [==============================] - 161s 433ms/step - loss: 0.0435 - accuracy: 0.9852 - val_loss: 0.0425 - val_accuracy: 0.9858 Epoch 00015: val_accuracy did not improve from 0.98868
Save the model¶
In [ ]:
model.save('My_Model.h5')
Test our model¶
In [ ]:
test_loss, test_accuracy = model.evaluate(Validation_generator)
test_loss = round(test_loss, 7)
test_accuracy = round(test_accuracy*100, 3)
print('Test Loss: ', test_loss)
print('Test Accuracy:', '\033[0m', test_accuracy,'%\033[0m')
93/93 [==============================] - 13s 130ms/step - loss: 0.0425 - accuracy: 0.9858 Test Loss: 0.0425473 Test Accuracy: 98.581 %
In [ ]:
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Model Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left', bbox_to_anchor=(1,1))
plt.show()
In [ ]:
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('Model Accuracy')
plt.ylabel('AUC')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left', bbox_to_anchor=(1,1))
plt.show()
Tensorboard¶
In [ ]:
%load_ext tensorboard
%tensorboard --logdir '/content/logs'
Model - 2¶
In [ ]:
def human_action_recognition_model_1():
input_layer = tf.keras.Input(shape=(Image_size,Image_size,3))
#First-Block
conv_1 = tf.keras.layers.Conv2D(filters=64,
kernel_size=(3,3),
padding='same',
activation='relu')(input_layer)
batchnorm_1 = tf.keras.layers.BatchNormalization()(conv_1)
dropout_1 = tf.keras.layers.Dropout(0.1)(batchnorm_1)
conv_2 = tf.keras.layers.Conv2D(filters=64,
kernel_size=(3,3),
padding='same',
activation='relu')(dropout_1)
batchnorm_2 = tf.keras.layers.BatchNormalization()(conv_2)
dropout_2 = tf.keras.layers.Dropout(0.1)(batchnorm_2)
maxpool_1 = tf.keras.layers.MaxPool2D(pool_size=(2,2))(dropout_2)
#Second-Block
conv_3 = tf.keras.layers.Conv2D(filters=128,
kernel_size=(3,3),
padding='same',
activation='relu')(maxpool_1)
batchnorm_3 = tf.keras.layers.BatchNormalization()(conv_3)
dropout_3 = tf.keras.layers.Dropout(0.1)(batchnorm_3)
conv_4 = tf.keras.layers.Conv2D(filters=128,
kernel_size=(3,3),
padding='same',
activation='relu')(dropout_3)
batchnorm_4 = tf.keras.layers.BatchNormalization()(conv_4)
dropout_4 = tf.keras.layers.Dropout(0.1)(batchnorm_4)
maxpool_2 = tf.keras.layers.MaxPool2D(pool_size=(2,2))(dropout_4)
#Third-Block
conv_5 = tf.keras.layers.Conv2D(filters=256,
kernel_size=(3,3),
padding='same',
activation='relu')(maxpool_2)
batchnorm_5 = tf.keras.layers.BatchNormalization()(conv_5)
dropout_5 = tf.keras.layers.Dropout(0.1)(batchnorm_5)
conv_6 = tf.keras.layers.Conv2D(filters=256,
kernel_size=(3,3),
padding='same',
activation='relu')(dropout_5)
batchnorm_6 = tf.keras.layers.BatchNormalization()(conv_6)
dropout_6 = tf.keras.layers.Dropout(0.1)(batchnorm_6)
conv_7 = tf.keras.layers.Conv2D(filters=256,
kernel_size=(3,3),
padding='same',
activation='relu')(dropout_6)
batchnorm_7 = tf.keras.layers.BatchNormalization()(conv_7)
dropout_7 = tf.keras.layers.Dropout(0.1)(batchnorm_7)
maxpool_3 = tf.keras.layers.MaxPool2D(pool_size=(2,2))(dropout_7)
#Forth Block
conv_8 = tf.keras.layers.Conv2D(filters=512,
kernel_size=(3,3),
padding='same',
activation='relu')(maxpool_3)
batchnorm_8 = tf.keras.layers.BatchNormalization()(conv_8)
dropout_8 = tf.keras.layers.Dropout(0.1)(batchnorm_8)
conv_9 = tf.keras.layers.Conv2D(filters=512,
kernel_size=(3,3),
padding='same',
activation='relu')(dropout_8)
batchnorm_9 = tf.keras.layers.BatchNormalization()(conv_9)
dropout_9 = tf.keras.layers.Dropout(0.1)(batchnorm_9)
conv_10 = tf.keras.layers.Conv2D(filters=512,
kernel_size=(3,3),
padding='same',
activation='relu')(dropout_9)
batchnorm_10 = tf.keras.layers.BatchNormalization()(conv_10)
dropout_10 = tf.keras.layers.Dropout(0.1)(batchnorm_10)
maxpool_4 = tf.keras.layers.MaxPool2D(pool_size=(2,2))(dropout_10)
#Fifth Block
conv_11 = tf.keras.layers.Conv2D(filters=512,
kernel_size=(3,3),
padding='same',
activation='relu')(maxpool_4)
batchnorm_11 = tf.keras.layers.BatchNormalization()(conv_11)
dropout_11 = tf.keras.layers.Dropout(0.1)(batchnorm_11)
conv_12 = tf.keras.layers.Conv2D(filters=512,
kernel_size=(3,3),
padding='same',
activation='relu')(dropout_11)
batchnorm_12 = tf.keras.layers.BatchNormalization()(conv_12)
dropout_12 = tf.keras.layers.Dropout(0.1)(batchnorm_12)
conv_13 = tf.keras.layers.Conv2D(filters=512,
kernel_size=(3,3),
padding='same',
activation='relu')(dropout_12)
maxpool_5 = tf.keras.layers.MaxPool2D(pool_size=(2,2))(conv_13)
flatten_layer = tf.keras.layers.Flatten()(maxpool_5)
dense_1 = tf.keras.layers.Dense(1024, activation='relu',kernel_initializer='he_uniform')(flatten_layer)
batchnorm_1 = tf.keras.layers.BatchNormalization()(dense_1)
dropout_1 = tf.keras.layers.Dropout(0.3)(batchnorm_1)
dense_2 = tf.keras.layers.Dense(512, activation='relu',kernel_initializer='he_uniform')(dropout_1)
batchnorm_2 = tf.keras.layers.BatchNormalization()(dense_2)
dropout_2 = tf.keras.layers.Dropout(0.5)(batchnorm_2)
dense_3 = tf.keras.layers.Dense(256, activation='relu',kernel_initializer='he_uniform')(dropout_2)
batchnorm_3 = tf.keras.layers.BatchNormalization()(dense_3)
dropout_3 = tf.keras.layers.Dropout(0.3)(batchnorm_3)
dense_4 = tf.keras.layers.Dense(256, activation='relu',kernel_initializer='he_uniform')(dropout_3)
batchnorm_4 = tf.keras.layers.BatchNormalization()(dense_4)
dropout_4 = tf.keras.layers.Dropout(0.5)(batchnorm_4)
output_layer = tf.keras.layers.Dense(output_size, activation='softmax')(dropout_4)
action_model_1 = tf.keras.Model(inputs=input_layer, outputs=output_layer)
return action_model_1
In [ ]:
model_1 = human_action_recognition_model_1()
In [ ]:
model_1.summary()
Model: "model" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(None, 224, 224, 3)] 0 _________________________________________________________________ conv2d (Conv2D) (None, 224, 224, 64) 1792 _________________________________________________________________ batch_normalization (BatchNo (None, 224, 224, 64) 256 _________________________________________________________________ dropout (Dropout) (None, 224, 224, 64) 0 _________________________________________________________________ conv2d_1 (Conv2D) (None, 224, 224, 64) 36928 _________________________________________________________________ batch_normalization_1 (Batch (None, 224, 224, 64) 256 _________________________________________________________________ dropout_1 (Dropout) (None, 224, 224, 64) 0 _________________________________________________________________ max_pooling2d (MaxPooling2D) (None, 112, 112, 64) 0 _________________________________________________________________ conv2d_2 (Conv2D) (None, 112, 112, 128) 73856 _________________________________________________________________ batch_normalization_2 (Batch (None, 112, 112, 128) 512 _________________________________________________________________ dropout_2 (Dropout) (None, 112, 112, 128) 0 _________________________________________________________________ conv2d_3 (Conv2D) (None, 112, 112, 128) 147584 _________________________________________________________________ batch_normalization_3 (Batch (None, 112, 112, 128) 512 _________________________________________________________________ dropout_3 (Dropout) (None, 112, 112, 128) 0 _________________________________________________________________ max_pooling2d_1 (MaxPooling2 (None, 56, 56, 128) 0 _________________________________________________________________ conv2d_4 (Conv2D) (None, 56, 56, 256) 295168 _________________________________________________________________ batch_normalization_4 (Batch (None, 56, 56, 256) 1024 _________________________________________________________________ dropout_4 (Dropout) (None, 56, 56, 256) 0 _________________________________________________________________ conv2d_5 (Conv2D) (None, 56, 56, 256) 590080 _________________________________________________________________ batch_normalization_5 (Batch (None, 56, 56, 256) 1024 _________________________________________________________________ dropout_5 (Dropout) (None, 56, 56, 256) 0 _________________________________________________________________ conv2d_6 (Conv2D) (None, 56, 56, 256) 590080 _________________________________________________________________ batch_normalization_6 (Batch (None, 56, 56, 256) 1024 _________________________________________________________________ dropout_6 (Dropout) (None, 56, 56, 256) 0 _________________________________________________________________ max_pooling2d_2 (MaxPooling2 (None, 28, 28, 256) 0 _________________________________________________________________ conv2d_7 (Conv2D) (None, 28, 28, 512) 1180160 _________________________________________________________________ batch_normalization_7 (Batch (None, 28, 28, 512) 2048 _________________________________________________________________ dropout_7 (Dropout) (None, 28, 28, 512) 0 _________________________________________________________________ conv2d_8 (Conv2D) (None, 28, 28, 512) 2359808 _________________________________________________________________ batch_normalization_8 (Batch (None, 28, 28, 512) 2048 _________________________________________________________________ dropout_8 (Dropout) (None, 28, 28, 512) 0 _________________________________________________________________ conv2d_9 (Conv2D) (None, 28, 28, 512) 2359808 _________________________________________________________________ batch_normalization_9 (Batch (None, 28, 28, 512) 2048 _________________________________________________________________ dropout_9 (Dropout) (None, 28, 28, 512) 0 _________________________________________________________________ max_pooling2d_3 (MaxPooling2 (None, 14, 14, 512) 0 _________________________________________________________________ conv2d_10 (Conv2D) (None, 14, 14, 512) 2359808 _________________________________________________________________ batch_normalization_10 (Batc (None, 14, 14, 512) 2048 _________________________________________________________________ dropout_10 (Dropout) (None, 14, 14, 512) 0 _________________________________________________________________ conv2d_11 (Conv2D) (None, 14, 14, 512) 2359808 _________________________________________________________________ batch_normalization_11 (Batc (None, 14, 14, 512) 2048 _________________________________________________________________ dropout_11 (Dropout) (None, 14, 14, 512) 0 _________________________________________________________________ conv2d_12 (Conv2D) (None, 14, 14, 512) 2359808 _________________________________________________________________ max_pooling2d_4 (MaxPooling2 (None, 7, 7, 512) 0 _________________________________________________________________ flatten (Flatten) (None, 25088) 0 _________________________________________________________________ dense (Dense) (None, 1024) 25691136 _________________________________________________________________ batch_normalization_12 (Batc (None, 1024) 4096 _________________________________________________________________ dropout_12 (Dropout) (None, 1024) 0 _________________________________________________________________ dense_1 (Dense) (None, 512) 524800 _________________________________________________________________ batch_normalization_13 (Batc (None, 512) 2048 _________________________________________________________________ dropout_13 (Dropout) (None, 512) 0 _________________________________________________________________ dense_2 (Dense) (None, 256) 131328 _________________________________________________________________ batch_normalization_14 (Batc (None, 256) 1024 _________________________________________________________________ dropout_14 (Dropout) (None, 256) 0 _________________________________________________________________ dense_3 (Dense) (None, 256) 65792 _________________________________________________________________ batch_normalization_15 (Batc (None, 256) 1024 _________________________________________________________________ dropout_15 (Dropout) (None, 256) 0 _________________________________________________________________ dense_4 (Dense) (None, 11) 2827 ================================================================= Total params: 41,153,611 Trainable params: 41,142,091 Non-trainable params: 11,520 _________________________________________________________________
In [ ]:
plot_model(model_1, show_shapes=True, show_layer_names=True, to_file='Model_1.png')
Out[ ]:
Compile the Model¶
In [ ]:
model_1.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
loss=tf.keras.losses.SparseCategoricalCrossentropy(),
metrics=['accuracy'])
Callbacks¶
In [ ]:
from tensorflow.keras.models import load_model
model_1 = load_model('/content/drive/MyDrive/Technocolabs_Internship_Project/ModelCheckpoint/best_weights_1.hdf5')
In [ ]:
model_checkpoint = tf.keras.callbacks.ModelCheckpoint(filepath='/content/ModelCheckpoint/best_weights_1.hdf5',
save_best_only=True,
monitor='val_accuracy',
mode='max',
verbose=1)
# tensorboard = tf.keras.callbacks.TensorBoard(log_dir='/content/logs')
Let's start the training¶
In [ ]:
history_1 = model_1.fit(Train_generator,
epochs=15,
batch_size=Batch_size,
validation_data=Validation_generator,
callbacks=[model_checkpoint])
Epoch 1/15 370/370 [==============================] - 228s 612ms/step - loss: 0.0305 - accuracy: 0.9849 - val_loss: 0.1649 - val_accuracy: 0.9498 Epoch 00001: val_accuracy improved from -inf to 0.94983, saving model to /content/ModelCheckpoint/best_weights_1.hdf5 Epoch 2/15 370/370 [==============================] - 225s 606ms/step - loss: 0.0256 - accuracy: 0.9863 - val_loss: 0.1246 - val_accuracy: 0.9691 Epoch 00002: val_accuracy improved from 0.94983 to 0.96909, saving model to /content/ModelCheckpoint/best_weights_1.hdf5 Epoch 3/15 370/370 [==============================] - 225s 607ms/step - loss: 0.0218 - accuracy: 0.9880 - val_loss: 0.2876 - val_accuracy: 0.9399 Epoch 00003: val_accuracy did not improve from 0.96909 Epoch 4/15 370/370 [==============================] - 225s 606ms/step - loss: 0.0336 - accuracy: 0.9832 - val_loss: 0.4115 - val_accuracy: 0.9015 Epoch 00004: val_accuracy did not improve from 0.96909 Epoch 5/15 370/370 [==============================] - 225s 607ms/step - loss: 0.0299 - accuracy: 0.9854 - val_loss: 0.1615 - val_accuracy: 0.9772 Epoch 00005: val_accuracy improved from 0.96909 to 0.97720, saving model to /content/ModelCheckpoint/best_weights_1.hdf5 Epoch 6/15 370/370 [==============================] - 226s 608ms/step - loss: 0.0231 - accuracy: 0.9874 - val_loss: 0.4197 - val_accuracy: 0.9748 Epoch 00006: val_accuracy did not improve from 0.97720 Epoch 7/15 370/370 [==============================] - 226s 608ms/step - loss: 0.0257 - accuracy: 0.9861 - val_loss: 0.1295 - val_accuracy: 0.9804 Epoch 00007: val_accuracy improved from 0.97720 to 0.98041, saving model to /content/ModelCheckpoint/best_weights_1.hdf5 Epoch 8/15 370/370 [==============================] - 226s 608ms/step - loss: 0.0213 - accuracy: 0.9875 - val_loss: 0.1713 - val_accuracy: 0.9514 Epoch 00008: val_accuracy did not improve from 0.98041 Epoch 9/15 370/370 [==============================] - 226s 608ms/step - loss: 0.0242 - accuracy: 0.9868 - val_loss: 0.3218 - val_accuracy: 0.9098 Epoch 00009: val_accuracy did not improve from 0.98041 Epoch 10/15 370/370 [==============================] - 225s 607ms/step - loss: 0.0217 - accuracy: 0.9879 - val_loss: 0.0474 - val_accuracy: 0.9829 Epoch 00010: val_accuracy improved from 0.98041 to 0.98294, saving model to /content/ModelCheckpoint/best_weights_1.hdf5 Epoch 11/15 370/370 [==============================] - 226s 608ms/step - loss: 0.0243 - accuracy: 0.9864 - val_loss: 0.0404 - val_accuracy: 0.9856 Epoch 00011: val_accuracy improved from 0.98294 to 0.98564, saving model to /content/ModelCheckpoint/best_weights_1.hdf5 Epoch 12/15 370/370 [==============================] - 225s 607ms/step - loss: 0.0222 - accuracy: 0.9866 - val_loss: 0.0248 - val_accuracy: 0.9904 Epoch 00012: val_accuracy improved from 0.98564 to 0.99037, saving model to /content/ModelCheckpoint/best_weights_1.hdf5 Epoch 13/15 370/370 [==============================] - 225s 607ms/step - loss: 0.0178 - accuracy: 0.9887 - val_loss: 0.0306 - val_accuracy: 0.9905 Epoch 00013: val_accuracy improved from 0.99037 to 0.99054, saving model to /content/ModelCheckpoint/best_weights_1.hdf5 Epoch 14/15 370/370 [==============================] - 226s 608ms/step - loss: 0.0207 - accuracy: 0.9887 - val_loss: 0.1213 - val_accuracy: 0.9615 Epoch 00014: val_accuracy did not improve from 0.99054 Epoch 15/15 370/370 [==============================] - 225s 608ms/step - loss: 0.0188 - accuracy: 0.9886 - val_loss: 0.0882 - val_accuracy: 0.9899 Epoch 00015: val_accuracy did not improve from 0.99054
In [ ]: